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Description 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0001] The present invention relates to an image 
processing apparatus. More particularly, the present in- 
vention relates to an Image processing apparatus for de- 
tecting a main subject from image data. 

2. Description of the Related Art 

[0002] There has been conventionally known various 
Icinds of arts for photographing desired pictures which 
find a main subject from the other subjects, adjust the 
timing of a photograph based on the condition of the 
main subject and select an appropriate image from a 
plurality of photographed images. 
[0003] The general method of finding a main subject 
from the other subjects Is to detect a main subject 
through finding a distinctive feature as a main subject in 
an image. In a case to detect a person from an image, 
for example, eyes or a mouth are recognized as distinc- 
tive feature. 

[0004] However, It takes much time to find a main sub- 
ject througli all the areas of an Image. Even if it is the 
case to find eyes or a mouth, there may be sometimes 
a case that the other subjects have similar brightness to 
that of the eyes or the nnouth to be detected. In this case, 
it may be possible that a camera misldentlfies as a main 
subject. In particular, there is a high likelihood to mlsi- 
dentify objects if those objects are seen in a row, even 
if the objects actually stand at different distances from 
one another. 

SUMMARY OF THE INVENTION 

[0005] Therefore, It Is an obj ect of the present inven- 
tion to provide an image processing apparatus, an im- 
age processing method and recording medium, which 
are capable of overcoming the above drawbacks ac- 
companying the conventional art. The above and other 
objects can be achieved by combinations described In 
the independent claims. The dependent claims define 
further advantageous and exemplary combinations of 
the present invention. 

[0006] According to the first atspect of the present in- 
vention, an image processing apparatus includes: an 
image data input unit for inputting image data of an ob- 
ject; a range setting unit for setting a partial searching 
range, which defines a partial depth length in the image 
data; andapartial image extracting unit for extracting 
from the image data, the certain part of the object in- 
cluded in the partial searching range as a partial image. 
[0007] The Image data input unit may Include a means 
for inputting a parallactic image, which picks up an im- 
age of the object from a plurality of different perspec- 



tives. The image processing apparatus may further In- 
clude a depth distribution Infomnation receiving unit for 
receiving the depth distribution information based on the 
parallactic image. 

5 [0008] The Image processing apparatus may further 
include a main subject detecting unit for detecting a 
main subject fromthe partial image and receh/ing or ac- 
quiring main subject information. 
[0009] The main subject detecting unit may Include a 

10 distinctive parts detecting unit for detecting from the par- 
tial image, a distinctive part, which should be Included 
in the main subject; and an infomnation receiving unit for 
receiving the main subject Information based on the po- 
sition of the distinctive part detected by the distinctive 

15 parts detecting unit. 

[001 0] The range setting unit may Include a means for 
setting a first partial searching range and a means for 
setting a second partial searching range, which differs 
from the first partial searching range. The partial image 

20 extracting unit includes a means for extracting the part 
of the object included in the first partial searching range 
as a partial image andameans for extracting the part of 
the object included in the second partial searching range 
as a second partial image. Further, the main subject de- 

25 tecting unit includes a means for detecting the main sub- 
ject from the first partial image and a means for detecting 
the main subject from the second partial image when 
the main subject is not detected from the first partial Im- 
age. 

30 [0011] The range setting unit may set a predeter- 
mined depth length as the first partial searching range 
and sets the depth length contiguous to the first partial 
searching range as the second partial searching range. 
[0012] The range setting unit sets the depth length, 

35 which includes an independent object existing at the 
nearest distance, as the first partial searching range 
based on the depth distribution information and sets the 
depth length contiguous to the first partial searching 
range as the second partial searching range. 

40 [0013] The range setting unit may divide the depth 
range that Includes the object in the image data based 
on the depth distribution information and sets a plurality 
of the partial searching ranges. The partial image ex- 
tracting unit may extract from the image data, the part 

45 of the object included in each of a plurality of the partial 
searching ranges as a plurality of the partial image. In 
addition, the main subject detecting unit may detect the 
main subject from each of a plurality of the partial image 
and receives a plurality of the main subject information. 

50 [0014] The range setting unit may set a different the 
partial searching range when the main subj ect Is not 
detected in the partial searching range set by the range 
setting unit. The partial image extracting unit may ex- 
tract again the partial Image. The main subject detecting 

55 unit may detect again the main subject. 

[0015] The Image data Input unit may be an Image 
capturing unit for picking up an Image of the object. The 
image processing apparatus may include a photograph- 
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ic condition deciding unit for deciding a photographic 
condition based on the main subject Information, and an 
image capturing controi unit for controlling imaging by 
the Image capturing unit based on the photographic con- 
dition. 

[0016] The image processing apparatus may further 
include a processing condition deciding unit for deciding 
an image processing condition based on the main sub- 
ject information and an image processor for processing 
the image data based on the image processing condi- 
tion. 

[0017] The image processing apparatus may further 
include an output condition deciding unit for deciding an 
image outputting condition based on the main subject 
infonnation and an image data output unit for outputting 
the image data based on the Image outputting condition. 
[0018] The image processing apparatus may further 
include a selecting condition storing unit for storing a 
predetermined selecting condition of which the main 
subject should satisfy, and an image selecting unit for 
selecting from a plurality of the image data input by the 
Image data input unit, a desired Image data, which in- 
cludes a desired main subject satisfying the selecting 
condition. 

[0019] The image data input unit Is an image captur- 
ing unit, which captures an image of the object based 
on a photographic timing signal. The image processing 
apparatus may include a timing condition storing unit for 
storing a predetermined photographic timing condition 
relating to the main subject, and a timing signal output 
unit for outputting the photographic timing signal to the 
image capturing unit when the main subject satisfies the 
photographic timing condition. 

[0020] An Image processing method according to the 
present invention includes: inputting image data of an 
object; setting a partial searching range, which defines 
a partial depth length for searching a main subject in the 
image data; extracting from the image data, the part of 
an object Included in the partial searching range as a 
partial image, based on a depth distribution information 
showing the distance between a camera and each part 
of the object included in the Image data; detecting the 
main subject in the partial image; and receiving a pre- 
determined main subject information relating to the de- 
tected main subject 

[0021] The setting Is a first setting of a first partial 
searching range as the partial searching range, the ex- 
tracting is a first extracting of the part of the object in- 
cluded in the first partial searching range, as a first par- 
tial image, and the detecting is a first detecting of the 
main subject in the first partial Image. The image 
processing method may Include a second setting of a 
second partial searching range as the partial searching 
range when the main subject is not detected in the first 
partial image, a second extracting of the part of the ob- 
ject included in the second partial searching range as a 
second partial image, and a second detecting of the 
main subject in the second partiei! image. 



[0022] The first setting is a setting of the depth length 
that Includes an Independent object existing at the near- 
est distance as the first partial searching range based 
on the depth distribution infomnation, while the second 
5 setting is a setting of the depth length contiguous to the 
first partial searching range, as the second partial 
searching range. 

[0023] The setting Is a dividing the depth range that 
includes the object In the Image data based on the depth 
10 distribution infomnation and a setting of a plurality of the 
partial searching range. The extracting is a extracting 
from the image data, of the part of the object Included 
in each of a plurality of the partial searching range as a 
plurality of the partial image. The detecting is a detecting 
IS of the main subject from each of a plurality of the partial 
image. Further, the receiving is a receiving of the main 
subject information relating to each of a plurality of the 
detected partial searching range. 
[0024] The inputting is an inputting at an image cap- 
20 turing unit, which captures an image of the obj ect. The 
image processing method may include deciding a pho- 
tographic condition based on the main subject informa- 
tion, and controlling an Image capturing at the image 
capturing unit based on the photographic condition. 
25 [0025] The Image processing method may further in- 
clude deciding an Image processing condition based on 
the main subject Infomnation, and processing the image 
data based on the image processing condition. 
[0026] The image processing method may further in- 
30 elude deciding an image outputting condition based on 
the main subject information and outputting the image 
data based on the image outputting condition. 
[0027] The image processing method may further in- 
clude storing a predetemnined selecting condition relat- 
es Ing to the main subject, and selecting from a plurality of 
the Image data input at the Inputting, a desired image 
data that includes a desired main subject satisfying the 
selecting condition. 

[0028] The inputting Is a capturing of an image of the 

40 object based on a photographic timing signal. The im- 
age processing method may further include storing a 
predetermined photographic timing condition relating to 
the main subject, and outputting the photographic timing 
signal to the Image capturing unit when the main subject 

45 satisfies the photographic timing condition. 

[0029] A recording medium, which records a program 
for processing an image and can be read by a computer, 
includes: a module for inputting image data of an object; 
a module for setting a partial searching range, which de- 

50 fines the partial depth length for searching a main sub- 
ject in the image data; a module for extracting from the 
image data, the part of the object included in the partial 
searching range as a partial Image, based on depth dis- 
tribution Information showing the distance between a 

55 camera and the object included in the image data; a 
module for detecting the main subject from the partial 
image; and a module for receiving predetermined main 
subject information relating to detected main subject. 
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[0030] This summary of the Invention does not nec- 
essarily describe all necessary features of the present 
invention. The present invention may also be a sub- 
combination of the above descnbed features. The 
above and other features and advantages of the present 
invention will become more apparent from the following 
description of ennbodiments talcen in conjunction with 
the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0031 ] Rg. 1 shows a structural view of a digital cam- 
era. 

[0032] Fig. 2 Is a functional block diagram of a digital 
camera according to the first embodiment 
[0033] Fig. 3 is a flowchart showing an image 
processing method according to the first embodiment. 
[0034] Rg. 4 is a flowchart showing an image 
processing method according to the second embodi- 
ment 

[0035] Rg. 5 is a functional block diagram of a digital 

camera according to the third embodiment 

[0036] Fig. 6 shows a structural view of a laboratory 

system. 

[0037] Fig. 7 is a functional biock diagram of a labo- 
ratory system according to the fourth embodiment. 
[0038] Fig. B is a functional block diagram of a labo- 
ratory system according to the fifth embodiment. 
[0039] Rg. 9 is a functional block diagram of a labo- 
ratory system according to the sixth embodiment. 
[0040] Fig. 1 0 Is a block diagram showing the struc- 
ture of a program according to the seventh embodiment 
[0041] Fig. 11 is a schematic view showing a distance 
measurement system of an active type according to a 
first example. 

[0042] Fig. 1 2 Is a schematic top view showing a dis- 
tance measurement system of an active type according 
to a second example. 

[0043] Rg. 1 3 is a schematic top view showing a dis- 
tance measurement system of an acth^e type according 
to an arrangement of the second example illustrated In 
Fig. 12. 

[0044] Fig. 1 4 is a chart for explaining the concept of 
the distance measuring system according to a third ex- 
ample of the present invention. 

[0045] Rg. 15 is afunctional block diagram of the dig- 
ital camera according to the eighth embodiment of the 
invention. 

DETAILED DESCRIPTION OF THE INVENTION 

[0046] The invention wiil now be described based on 
the preferred embodiments, which do not intend to limit 
the range of the present Invention, but exemplify the in- 
vention. All of the features and the combinations thereof 
described in the embodiment are not necessarily essen- 
tial to the invention. 

[0047] The image processing apparatus of the 



present Invention described In the following can be 
achieved as a digital camera or a laboratory system, for 
example, In each of the following embodiments. A digital 
still camera or a digital video camera, for example, are 

5 included in the digital camera. 

[0046] The first embodiment of the present invention 
wiil be described in the following. The image processing 
apparatus according to the present embodiment is a dig- 
ital camera, which detects a main subject and takes a 

10 photograph under the best photographing condition. A 
main subject here means an Independent object, which 
is intentionally photographed amongst the photograph- 
ing objects. For example, a main subject will be some- 
thing such as a person who is photographed in a room, 

IS fish that is photographed swimming in an aquarium, or 
a bird that Is photographed perching on a branch of a 
tree. 

[0049] Fig. 1 is a block diagram depicting a structure 
of a digital camera 1 0 according to the preferred embod- 

20 iment of the Invention. As shown in Fig. 1 , a digital cam- 
era 10 is mainly comprised of an image capturing unit 
20, an image capturing control unit 40, a processing unit 
60, a display unit 100, and an operating unit 110. 
[0050] The image capturing unit 20 Includes structural 

25 and electrical members pertaining to photographing and 
image fonnlng. That is, Image capturing unit 20 is pro- 
vided with a lens section 22, which captures an image 
and processes the captured image, a diaphragm 24, a 
shutter 26, an optical LPF or low pass filter 28, a CCD 

30 30, and a capture-signal processor 32. The lens section 
22 preferably includes a focus lens and a zoom lens. 
The structure makes it possible to form an image of an 
object on the surface of the light reception of the CCD 
30. Each sensor element of the CCD 30, though it Is not 

35 shown in Fig. 1 , stores up electric charge (referred to as 
"stored electric charge") . The stored electric charge Is 
taken up with a read gate pulse to a shift register not 
shown in Fig. 1 , and subsequently taken out sequential- 
ly with a register transfer pulse as a voltage signal. 

40 [0051] Since the digital camera 10 generally has an 
electrk) shutter function, a mechanical shutter such as 
the shutter 26 is not always necessary. A shutter drain 
Is set up in the CCD 30 through a shutter gate in order 
to achieve the electric shutter function. When the shutter 

45 gate is activated, the stored electric charge is drained 
out to the shutter drain. Shutter speed, which Is the time 
to store up the electric charge in each sensor element, 
can be controlled by controlling the shutter gate. 
[0052] The voltage signal output from the CCD 30, 

50 that is, an analog signal, is color-separated by the cap- 
ture-signal processor 32 Into R, G and B elements, and 
the white balance is adjusted. The capture-signal proc- 
essor 32 then takes place a gamma correction, sequen- 
tially converts the R, GandB signals from analog into 

55 digital form by the necessary timing, and outputs the ob- 
tained digital image data (refen-ed to as "digital image 
data") to the processing unit 60. 
[0053] The image capturing unit 20 further includes a 
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finder 34 and a electronic flash 36. The finder 34 may 
be equipped with an LCD not shown In Fig. 1 . In that 
case, the finder 34 displays various information provid- 
ed by a main CPU 62 described later. The electronic 
flEish 36 performs to emit light when the energy stored 5 
in a capacitor, not shown in Fig. 1 , is supplied to a dis- 
charge tube 36a. 

[0054] The image capturing control unit 40 Is com- 
prised of a zoom lens driver 42, a focus driver 44, a di- 
aphragm driver 46, a shutter driver 48, an image cap- 
turing system CPU 50, which controls all of the €ibove- 
mentioned drivers, a distance sensor 52, and a photom- 
etry sensor 54. Each of the drivers such as the zoom 
lens driver 42 has a driving means such as a stepping 
motor or the like. In a response to a snap of a release 
switch 1 1 4 described later, the distance sensor 52 gaug- 
es the distance defined between the camera and the ob- 
ject, and the photometry sensor 54 measures the bright- 
ness of the object. The gauged distance data (referred 
to as "gauged distance data") and the measured bright- 
ness data of the object (referred to as "measured lumi- 
nance data") are transmitted to the image capturing sys- 
tem CPU 50. The image capturing system CPU 50 con- 
trols the zoom lens driver 42 and the focus driver 44 
based on the photograph information provided by a user 
such as the magnification of the zoom lens, and makes 
adjustments of the magnification of the zoom lens and 
focus for the lens section 22. 

[0055] The image capturing system CPU 50 deter- 
mines an aperture value and a shutter speed, based on 
the Integrated value of digital signals of RGB in one Im- 
age frame, which is called as AE information. According 
to the aperture value and the shutter speed thus deter- 
mined, the diaphragm driver 46 and the shutter driver 
48 respectively adjust the amount of aperture and then 
drives the shutter 26 to open and close. 
[0056] The image capturing system CPU 50 also con- 
trols the light intensity emitting from the electronic flash 
36 based on the measured brightness data, and simul- 
taneously adjusts the amount of aperture of the dia- 
phragm 24. When a user instmcts a camera to pick up 
an image, the CCD 30 starts to store up the electric 
charge. Then, the stored electric charge is output to the 
capture-signal processor 32 after the expiration of the 
shutter time calculated on the basis of the measured 
brightness data. 

[0057] The processing unit 60 is comprised of a main 
CPU 62, which controls the whole digital camera 10, es- 
pecially controlling the processing unit 60 itself, a mem- 
ory controller 64, which Is controlled by the main CPU 
62, a YC processor 70, an optional device controller 74, 
a compression/extension processor 78, and a commu- 
nication interface (l/F) section 80. The main CPU 62 
communicates necessary Information with the image 
capturing system CPU 50 by serial communication, for 
example. A clock generator gives an operating clock of 
the main CPU 62. The dock generator 88 also provides 
clocks of different frequencies respectively to the Image 



capturing system CPU 50 and the display unit 100. 
[0058] The main CPU 62 is also comprised of a char- 
acter generator 84 and a timer 86. The timer 86 is 
backed up by batteries and counts the time and date 
continuously. This count value gives the main CPU 62 
Information about the time and date of photograph and 
other time information. The character generator 84 gen- 
erates character information such as the time and date 
of photograph or a title. Those character Information will 
then appropriately be superimposed with a photograph- 
ic image. 

[0059] The memory controller 84 controls a non-vol- 
atile memory 66 and a main memory 68. The non-vola- 
tile memory 66 is comprised of components such as an 
EEPROM or electrically-erasable programmable ROM, 
and a flash memory. The non-volatile memory 66 stores 
data such as setting information set by a user or an ad- 
justment value before shipment, which should be kept 
even when the power of the digital camera 1 0 is turned 
off. The non-volatile memory 66 may store a boot pro- 
gram or a system program of the main CPU 62 if nec- 
essary. On the other hand, the main memory 68 may 
generally be a relatively inexpensive memory having a 
large capacity such as a DRAM. The main memory 68 
has: a frame memory function, which stores data output 
from the Image capturing unit 20; a systemmemory func- 
tion, which is loaded with various programs; and a work 
area function. The non-volatile memory 66 and the main 
memory 68 control to transmit data back and forth 
among the components at the inside and outside of the 
processing unit 60 via a main bus 82. 
[0060] The YC processor 70 undertakes a Y-to-C con- 
version on digital image data and generates a lumi- 
nance signal Y and a chrominance signal B-Y and R-Y 
The memory controller 64 temporarily stores the lumi- 
nance signal and the chrominance signal In the main 
memory 68. The compression/extension processor 78 
sequentially reads out the luminance signal and the 
chrominance signal from the main memory 68. The 
compression/ extension processor then compresses 
the luminance signal and the chrominance signal. A 
memory card, whbh is one kind of the optional device 
76, writes through the optional device controller 74 the 
compressed data described above (referred to as "com- 
pressed data"). 

[0061] The processing unit 60 further Includes an en- 
coder 72. The encoder 72 inputs a luminance signal and 
a chrominance signal. Then, the encoder 72 converts 
the luminance signal and the chrominance signal into 
video signals such as National Television System Com- 
mittee (NTSC) and Phase Altemation by Line (PAL) sig- 
nals. The encoder 72 outputs the converted video sig- 
nals through a video output temriinal 90. In order to gen- 
erate a video signal from the data recorded In the op- 
tional device 76, firstiy the data is transmitted to the com- 
pression/ extension processor 78 through the optional 
device controller 74. Then, the data, which has been 
subjected to a necessary process by the compression/ 
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extension processor 78, Is converted into a video signal 
by the encoder 72. 

[0062] Tlie optional device controller 74 undertakes a 
generation of necessary signals, a logic conversion, and 
a voitage conversion, between tiie main bus 82 and the 
optional device 76. The digitai camera 1 0 may support, 
for example, an I/O card, which Is a standard PCMCIA- 
compliant card, other than the above-described memory 
card, as an optional device 76. In this case, the optional 
device controller 74 may Include a bus control LSI for 
PCMCIA. 

[0063] The communication l/F section 80 controls 
such as a protocol conversion based on a communica- 
tion specification for which the digital camera 10 sup- 
ports, that is, a specification such as USB, RS-232C, or 
Ethernet, for example. The communication i/F section 
80 may be provided with a driver IC if necessary, and 
communicates through external devices including net- 
works and a connector 92. Other than those standard 
specifications, the communication t/F section can have 
a structure which exchanges data, through a particular 
interface, with an external device such as a printer, a 
l^araoke device, or a game device. 
[0064] The display unit 100 Is comprised of an LCD 
monitor 102 and an LCD panel 104. A monitor driver 
1 06 and a panel driver 1 08 respectively control the LCD 
monitor 102 and the LCD panel. The LCD monitor 102 
may be, for example, a 2-inch monitor, and is set in the 
back of a camera. The LCD monitor 102 displays the 
modes of the present photographs and plays, magnifi- 
cation of the zoom lens for photographing and playing, 
the amount of energy left in batteries, time and date, a 
display for setting a mode, and subject images. The LCD 
panel 104 may be, for example, a small monochrome 
LCD, and is set on the top of a camera. The LCD panel 
1 04 simply displays Infonnation such as the Image qual- 
ity described as FINE/NORMAL/BASIC, on/off of the 
electronic flash, the standard number of possible pho- 
tographs, the number of pixels, and the amount of power 
left in the batteries. 

[0065] The operating unit 110 is comprised of appa- 
ratuses and electric members, which are necessary for 
a user to set or instruct the operation or the mode of the 
operation of the digital camera 1 0. The power switch 1 1 2 
detemnines whether the power of the digital camera 10 
is turned on or off. The release switch 114 has a dual 
structure having the half push switch and the full push 
switch. For example, the half push switch locks AF and 
AE, and the full push switch captures a photographic 
image. After necessary processes such as the signal 
processing and the compression of data, the captured 
image will be stored In the main memory 68 or the op- 
tional device 76. The operating unit 1 1 0 may have a set- 
ting such as a rotary dial tor mode or a cross l<ey other 
than the power switch 112 and the release switch 114. 
The dial for mode and the cross key are collectively re- 
ferred to as a function setting section 116 as in Rg. 1. 
There are "file fomnat", "special effects', "photographic 



printing", "conflmri/ save", and "display switching", for 
example, as an operation or a function, which can be 
designated by using the operating unit 110. The zoom 
switch 11 8 Is operated to set the magnification of zoom- 
s ing. 

[0066] Fig. 2 is a functional block diagram of the digital 
camera 1 0 according to the present embodiment. The 
digital camera 1 0 comprises an image data Input unit 
200, a depth distribution Information receiving unit 202, 

10 a range setting unit 204, a partial image extracting unit 
206, a distinctive parts detecting unlt208, an information 
receiving unit 210, a photographing condition setting 
unit 220, and an Image capturing control unit 222. The 
Image data input unit 200 Inputs Image data of an object. 

IS The image data Input unit 200 corresponds to the image 
capturing unit 20 In Fig. 1 . The image data Input unlt200 
also inputs aparallactic image, which picks up an Image 
of an object from a plurality of different aspects. 
[0067] The depth distribution information receiving 

20 unit 202 receives information about distribution of depth 
showing the distance from the lens to each part of an 
object, based on a parallactic image received from the 
image data input unit 200. in other words, the depth dis- 
tribution information receiving unit 202 undertakes a 

25 process for deciding a corresponding point and calcu- 
lates a parallax amount. Then, the depth distribution In- 
fomnation receiving unit 202 extracts information about 
distribution of depth based on the calculated parallax 
amount. The process for extracting infonnation about 

30 distribution of depth based on a parallax amount can be 
undertaken by the trigonometrical survey, which is con- 
ventionally known. 

[0068] The range setting unit 204 sets a partial 
searching range, which decides a partial depth length 

35 for searching a main subject in image data. It is prefer- 
able that the partial searching range be set at a depth 
length, which has a high possibility for a main subject to 
exist. For example, since it is more likely that a main 
subject exists at a nearest place from a camera amongst 

40 all of the objects, the range setting unit 204 may set, 
based on Information about distribution of depth, the 
depth length that Includes an independent object exist- 
ed In the nearest place, as a partial searching range. 
[0069] If a main subject Is not detected from the set 

45 partial searching range, the range setting unit 204 sets 
a further partial searching range. The partial searching 
range set after the first setting is set at a different depth 
length than the first setting. For example, the range set- 
ting unit 204 firstly sets a depth length to the nearest 

50 distance, and sets the next depth length at a contiguous 
distance from the first setting. Then, the range setting 
unit 204 may slide a range for setting a depth length as 
a partial searching range, and small increments are far- 
ther made until a main subject Is detected. 

55 [0070] This sliding method makes it possible for a 
camera notto mlsidentlfy the target object placed at dif- 
ferent depths as one object in detecting a main subject 
since the sliding excludes objects at different depths 
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from a searching target. Further, since this sliding meth- 
od makes only a place of depth having a high possibility 
of existence of a main subject a searching target, the 
time for detecting a main subject can be shortened. Us- 
ing this method, the image processing apparatus ac- 
cording to the present embodiment can detect a main 
subject with a high degree of accuracy or effectively. 
[0071] The partial image extracting unit 206 extracts 
the part of the object Included in a partial searching 
range from an image data as a partial image, based on 
information about distribution of depth. For example, the 
objects at long distances are not imaged in the partial 
image, which extracted an object at a closest distance. 
The distinctive parts detecting unit 208 detects from a 
partial image a distinctive part that should be included 
In a main subj ecL A distinctive part noted here means 
a necessary part, which has a distinctive feature espe- 
cially something such as colors or shapes, from among 
other parts included In a main subject. For example, 
when a person is a main subject, the eyes or the mouth 
of the person is the distinctive part. It is preferable that 
a distinctive part have colors and shapes that objects 
other than the main subject do not have. For example, 
the color of a facial skin can be a target for detecting as 
a distincth^e part since things other than human beings 
rarely have any chance to have the same color. Moreo- 
ver, the eye color of human beings can also be a target 
for detecting as a distinctive part since tilings other than 
human beings rarely have any chance to have the same 
color. 

[0072] The Information receiving unit 21 0 receives the 
prescribed information of a main subject (referred to as 
"main subject information") based on the position of the 
distinctive part If the distinctive parts detecting unit 208 
detects a distinctive part from a partial image. There is 
distance information between a lens and a main subject 
and infomnation about luminance of a main subject, for 
example, as main subject infomriation. 
[0073] The photographing condition setting unit 220 
decides photographing conditions based on main sub- 
ject Information. For example, the photographing condi- 
tion setting unit 220 determines a focal distance based 
on distance information between the lens and a main 
subject. The photographing condition setting unit 220 al- 
so sets an aperture value and an exposure time or a 
shutter speed, based on luminance Information of a 
main subject. 

[0074] The image capturing control unit 222 controls 
the imaging by the image capturing unit 20 based on the 
photographing conditions set by the photographing con- 
dition setting unit 220. For example, the focal distance 
determined by the photographing condition setting unit 
220 controls a zoom and a focus. Further, the aperture 
value and the exposure time set by the photographing 
condition setting unit 220 control the diaphragm 24 and 
the shutter 26. 

[0075] An appropriate image can therefore be ob- 
tained by the apparatus or the camera according to the 



present embodiment since the camera takes a photo- 
graph in conditions suited to a main subject, which is 
detected with high precision. 

[0076] Refening now to Fig. 3, a flowchart showing 

5 an image processing method are shown according to 
the present embodiment. This flowchart especially 
shows the method for detecting a main subject. First, 
the image data input unit 200 inputs image data of an 
object at the step S1 00. The Image data input unit 200 

10 inputs parallactic image data at the step S102. Then, 
the depth distribution infomriation receiving unit 202 re- 
ceives infomnation about distribution of depth at the step 
SI 04. The range setting unit 204 then sets a partial 
searching range at the step SI 06. The partial image ex- 

15 tracting unit 206 extracts a partial image at the step 
S1 08. Then, the distinctive parts detecting unit 208 de- 
tects a distinctive part from a partial Image at the step 
S11 0. if a distinctive part is not detected, the range set- 
ting unit 204 sets a different partial searx:hing mnge. 

20 Then the steps from SI 06 through S1 10 are repeated 
until a distinctive part is detected. If a distinctive part is 
detected by the distinctive parts detecting unit 208, the 
Information receiving unit 210 receives main subject in- 
fomnation at the step S1 12. 

25 [0077] The present invention accord Ing to the second 
embodiment will now be explained in the following. The 
image processing apparatus according to the present 
embodiment is a digital camera, which detects a plurality 
of main subjects from image data. The digital camera 

30 according to the present embodiment has almost the 
same structure and functions as the digital camera ac- 
cording to the first embodiment. Since the structure of 
the digital camera according to the present embodiment 
is substantially the same as the structure and functions 

35 shown in Fig. 2, the explanation is omitted. According 
to the present embodiment, the range setting unit 204 
divides the depth length that includes all of the objects 
into a plurality of ranges. The range setting unit 204 then 
sets the divided depth length as searching ranges. The 

40 camera tries to detect a main subject from all of these 
searching ranges. 

[007B] Referring now to Fig. 4, a flowchart showing 
an image processing method Is shown according to the 
present embodiment. This flowchart especially shows a 
45 method for detecting a main subject. First, the image 
data input unit 200 inputs Image data of an object at the 
step S150 . The Image data input unit 200 inputs paral- 
lactic image data at the step SI 52. Then, the depth dis- 
tribution information receiving unit 202 receives infor- 
50 mation about distribution of depth at the step SI 54. 
[0079] The range setting unit 204 then dhvldes the 
depth length that includes objects in Image data Into a 
plurality of ranges. Then the range setting unit 204 sets 
the divided depth length as a plurality of partial search- 
es ing ranges at the step S1 56. The partial image extracting 
unit 206 extracts from Image data, the part included In 
each of a plurality of partial searching ranges as a plu- 
rality of partial image at the step SI 56. Then, the dis- 
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tinctive parts detecting unit 208 detects a distinctive part 
fronn each of a plurality of partial images at the step 
S160.The information receiving unit 210 receives main 
subject infonnatlon from each of a plurality of the partial 
Images where a distinctive part is detected, at the step 
S162. 

[0080] A camera therefore divides the range of the 
depth that includes all of the objects and sets partial 
searching ranges. Then the camera detects a main sub- 
ject from all of these partial searching ranges. There- 
fore, all of the main subjects can be divided up and de- 
tected without a misunderstanding when a plurality of 
main subjects are included in image data. 
[0081] Tlie present invention according to the third 
embodiment will now be explained in the following. The 
image processing apparatus according to the present 
embodiment is a digital camera, which detects a main 
subject and photographs in the best timing. Fig. 5 is a 
functional block diagram of a digital camera according 
to the present embodiment. Since the depth distribution 
Information receiving unit 202, the range setting unit 
204. the partial image extracting unit 206, the distinctive 
parts detecting unit 208, and the information receiving 
unit 210 in Fig. 5 have the same functions as the first 
and second embodiments, the explanation is omitted. 
[0082] The image data input unit 200 is an image cap- 
turing unit, which photographs an obj ect based on a 
photographic timing signal. The image data input unit 
200 con^esponds to the image capturing unit 20 In Fig. 
1. A timing condition storing unit 226 stores predeter- 
mined photographing conditions relating to a main sub- 
ject. For example, when a person is a main subject, con- 
ditions such as "not blinking", "looking straight at the 
camera", or "smiling" of the person can be a photograph- 
ing timing condition. 

[0083] The timing signal output unit 224 outputs a 
photographic timing signal to the image data input unit 
when a distinctive part satisfies the photographic timing 
condition. The judgment method forjudging if a distinc- 
tive part satisfies a photographic timing condition de- 
pends on the photographing conditions. For example, 
photographing conditions such as "not blinking" and 
"looking straight at the camera", the size of an eye, the 
shape of a mouth, orthe size of a mouth can be decision 
making materials. Whether or not these decision making 
materials satisfy the photographing conditions depend 
on the decision by given algorithms based on the em- 
pirbal rules for the respective photographing conditions. 
[0084] Further, whether or not the person as a main 
subject takes a prBdetemriined behavior can be a basis 
for deciding a photographic timing. For example, in a 
case where a person is set as a main subject, photo- 
graphing is undertaken when the distinctive part of the 
person shows a predetermined change. As a predeter- 
mined change, "opened the eyes after closing the eyes 
tor more than two seconds" or 'the eyeshot depicted a 
given trajectory" can be thought, for example. As these 
photographic conditions, it is preferable that behaviors 



and changes that a person never takes in front of an 
ordinary camera be set as photographic conditions. 
These conditions are used to prevent photographing an 
undeslred image by an object person accidentally taking 

5 the behaviors included in the photographic conditions. 
[0085] The judgment method for judging whether or 
notthe distinctive part satisfies aphotographk: condition 
differs depending on photographk? conditions. If the 
photographic conditions are "blinking" and "eyeshot", 

10 the color, the shape, and the size of eyes are supposed 
to be the judgment making factors. Whether or not a 
Judgment making factor satisfies a photographic condi- 
tion depends on a given algorithm based on empirical 
rules for the respective photographic conditions. 

15 [0086] The image data input unit 200 photographs an 
Image in a timing based on a photographic timing signal 
received from the timing signal output unit 224. 
[0087] Using this method, a desired image can be ob- 
tained since the time when a main subject, which is de- 

20 tected with high precision, becomes a desired condition 
is set as a photographic timing. Moreover, since a main 
subject can be detected effectively and quickly, a de- 
sired image can be obtained without missing a photo- 
graphic timing. 

25 [0086] The present invention according to the fourth 

embodiment will now be explained In the following. The 
image processing apparatus according to the present 
embodiment is a laboratory system, which selects from 
a plurality of images, an Image whose main subject is 

30 the best-looking image. Referring now to Fig. 6, a struc- 
tural diagram of a laboratory system 300 Is shown. This 
laboratory system mainly comprises an input unit 310, 
a processing unit 360, a print unit 330, a display unit 
340, and an operating unit 320. 

35 [0089] The input unit 310 comprises an area CCD 
scanner 312 and a line CCD scanner 314. The area 
CCD scanner 312 and the line CCD scanner 31 4 obtain 
image data by scanning the image on a film or a picture 
print. A scanner controller 316 controls the area CCD 

40 scanner 31 2 and the line CCD scanner 31 4. The input 
unit 31 0 outputs the obtained image data to the process- 
ing unit 360. 

[0090] The processing unit 360 comprises the whole 
laboratory system, especially a main CPU 362, which 

45 controls the processing unit 360 Itself, a memory con- 
troller 364, which is controlled by the main CPU 362, a 
YC processor 370, an optional device controller 374, a 
compression/extension processor 378, and a commu- 
nication interface or l/F section 380. The operating clock 

50 of the main CPU 362 is provided from a clock generator 
388. The clock generator 388 also provides clocks of 
different frequencies to the display unit 340. 
[0091] The main CPU 362 also provides a character 
generator 384 and a timer 386. The timer 386 Is backed 

55 up by batteries and counts the time and date continu- 
ously. The character generator 384 generates character 
infomnation such as the time and date of a photograph 
or a title. Then, this character information will appropri- 
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ately be combined with a photographic Image. 
[0092] The memory controHer 364 controls a non-vol- 
atile memory 366 and a main memory 368 . The non- 
votatlle memory 366 Is comprised of components such 
as an EEPROM or electrically-erasable programmable 
ROM, and a flash memory. The non-volatile memory 
366 stores data such as setting infomiation set by a user 
or an adjustment value before shipment, which should 
be kept even when the power of the laboratory system 
300 is off. The non-volatile memory 366 may store a 
boot program or a system program of the main CPU 362 
if necessary. On the other hand, the main memory 368 
generally comprises a comparably cheap memory hav* 
Ing a large capacity such as a DRAM. The main memory 
368 performs: a frame memory function, which stores 
Image data Input by the Input unit 31 0; a system memory 
function, which loads various programs; and a work area 
function. The non-volatile memory 366 and the main 
memory 368 control to exchange data between each 
pari of the inside and outside of the processing unit 360 
via the main bus 382. 

[0093] The YG processor 370 undertakes a YC con- 
version on digital image data and generates a lumi- 
nance signal Y and a chrominance signal B-Y and R-Y. 
The memory controller 364 temporarily stores the lumi- 
nance signal and the chrominance signal to the main 
memory 368. The compression/extension processor 
378 sequentially reads out the luminance signal and the 
chrominance signal from the main memory 368. The 
compression/extension processor then compresses the 
luminance signal and the chrominance signal. A mem- 
ory card, whbh is one kind of the optional device 376, 
writes the compressed data described above (referred 
to as "compressed data"), through the optional device 
controller 374. 

[0094] The processing unit 360 further comprises an 
encoder 372. The encoder 372 inputs a luminance sig- 
nal and a chrominance signal. The encoder 372 then 
converts the luminance signal and the chrominance sig- 
nal to video signals such as National Television System 
Committee (NTSC) and Phase Altematlon by Une (PAL) 
signals. The encoder 372 outputs the converted video 
signals from a video output terminal 390. In orderto gen- 
erate a video signal from the data recorded in the op- 
tional device 376, firstly the data is transmitted to the 
compression/extension processor 78 through the op- 
tional device controller 374. Then, the data, which has 
been subjected to a necessary process by the compres- 
sion/extension processor 378, Is converted into a video 
signal by the encoder 372. 

[0095] The optional device controller 374 undertakes 
a generation of necessary signals, a logic conversion, 
and a voltage conversion, between the main bus 362 
and the optional device 376. The laboratory system 300 
may support, for example, a floppy disc or an MO, other 
than the above-described memory card, as an optional 
device 76. In this case, the optional device controller 374 
may include a floppy disc drive or an MO drive. 



[0096] The communication l/F section 380 under- 
takes a control such as a protocol conversion based on 
a communication specification for which the laboratory 
system 300 supports, that is, a specification such as 

5 USB, RS-232C, or Ethernet, for example. The commu- 
nication l/F section 380 may be provided with a driver 
IC if necessary, and communicates through external de- 
vices including networks and a connector 392. 
[0097] The operating unit 360 further includes a hard 

10 disc 394 as a secondary storage unit The hard disc 394 
stores, for example, image data input by the input unit 
310, image data that has undertaken given image 
processing, data read from the optional device 376, and 
data Input through the communication l/F section 380. 

15 [0098] The processing unit 360 further includes a CD- 
ROM drive 396. The CD-ROM drive 396, for example, 
acthfates the processing unit 360 and picks up a pro- 
gram from the CD-ROM for operating the processing 
unit 360 to perfonm predetermined functions. 

^ [0099] The display unit 340 includes a monitor 342 
and a monitor driver 346, whteh controls the monitor 
342. A CRT monitor or an LCD monitor may be used as 
the monitor 342. 

[01 00] The operating unit 320 comprises apparatuses 
25 and electric mennbers, which are necessary for a user 
to set or Instruct the operation or the mode of the oper- 
ation of the laboratory system 300. The operating unit 
320 comprises, for example, a keyboard 322 and a 
mouse 324. 

30 [0101] The print unit 330 com prises a printer 332 and 
a printer controller 334, which controls the printer 332. 
There are, for example, a laser printer and an Inkjet 
printer as the printers 332. 

[0102] Fig. 7 Is a functional block diagram of the lab- 

35 oratory system according to the present embodiment. 
Since the depth distribution information receiving unit 
202, the range setting unit 204, the partial image extract- 
ing unit 206, the distinctive parts detecting unit 208. and 
the Information receiving unit 210 shown in Fig. 7 have 

40 the same functions as the first and second embodi- 
ments, the explanation is omitted. 
[01 03] The image data input unit 200 inputs image da- 
ta of an object. The image data input unit 200 corre- 
sponds, for example, to the Input unit 310, the optional 

45 device controller 374, the communication l/F section 
380, and the CD-ROM drive 396. When the image 
processing is undertaken by reading the image data 
from a film or a picture print, the area CCD scanner 31 2 
or the line CCD scanner 314 included in the input unit 

50 310 reads image data on the film or the picture print. 
When the image processing is undertaken by inputting 
image data recorded in a memory card, a card drive as 
the optional device controller 374 reads image data from 
a memory card as an example of the optional device 

55 376. When image is inputted to be processed, for ex- 
ample, a floppy diskette drive serving as the optional de- 
vice controller 374 reads out image data from a floppy 
diskette as an example of the optional device 376. It may 
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also be applicable In another arrangement of the Inven- 
tion, that an MO drive serving as the optional device con- 
troller 374 reads out image data from an MO diskette as 
an example of the optional device 376. 
[0104] When the image processing is undertaken by 
inputting Image data recorded on a CD-ROM 398, the 
CD-ROM 398 reads Image data from the CD-ROM drive 
396. When image to be processed is inputted through 
a network , Image data Is taken In through the commu- 
nication l/F section 3 BO. 

[0105] The selecting condition storing unit 232 stores 
predetermined selecting conditions relating to a main 
subject. The selecting conditions noted here means the 
conditions for selecting a desired image from a plurality 
of images. It is said that most of the objects photo- 
graphed by cameras are of people. Therefore, In select- 
ing a desired image from a plurality of Images, whether 
or not an object person is looking good can be a basis 
for selection most of the time. According to the present 
embodiment, the laboratory system 300 selects a de- 
sired image from a plurality of images whose objects are 
persons. "Not blinking", "not red-eyed", "looking straight 
to the camera", or "smiling" can be thought of as condi- 
tions for a person to be photographed while looking 
good. 

[0106] The judgment method for Judging whether or 
not the distinctive part satisfies a photographic condition 
differs depending on photographic conditions. If the 
photographic conditions are as to "not blinking", "not 
red-eyed", and "looking straight to the camera", the 
color, the shape, and the size of the eyes are supposed 
to be the judgment making factors, if the photographic 
conditions are as to "smiling", the size of the eyes, the 
shape of the mouth, and the size of the mouth are sup- 
posed to be the Judgment making factors. Whether or 
not these judgment making factors satisfy a photograph- 
ic condition depend on a given algorithm based on em- 
pirical rules for the respective photographic conditions. 
[0107] The image selecting unit 230 selects from a 
plurality of image data Input by the Image data input unit 
200, a desired Image data including a desired main sub- 
ject that satisfies the selecting conditions. In other 
words, the image selecting unit 230 judges whether or 
not the main subject detected by the information receiv- 
ing unit 210 satisfies the selecting conditions stored in 
the selecting condition storing unit 232. A desired image 
including a desired main subject that satisfies the se- 
lecting conditions will be stored on the hard disc 394 and 
be displayed on the monitor 342, in Fig.6. The optional 
device controller 374 may record desired images on a 
recording medium such as a memory card, a floppy disc, 
and an MO. The communication l/F section 380 may 
output a desired image through a network. 
[0108] According to another embodiment, this Image 
processing apparatus may be a digital camera, which 
selects from a plurality of images, an image having the 
best look of a main subject. In this case, the structure 
of the digital camera is as shown in Fig, 1 . 



[01 09] A desired image with high precision can there- 
fore be selected since a main subject detected with high 
precision is supposed to be a target for deciding when 
selecting an image. 

5 [0110] The present invention according to the fifth em- 
bodiment will now be explained. The image processing 
unit according to the present embodiment is a laboratory 
system, which detects a main subject and undertakes 
the appropriate Image processing. Since the structure 

10 of this laboratory system is the same as the fourth em- 
bodiment in Fig. 6, the explanation Is omitted. 
[01 1 1] Refening now to Fig. 8, a functional block dia- 
gram of the laboratory system is shown according to the 
present embodiment. Since the Image data input unit 

15 200, the depth distribution infomnation receiving unit 
202, the range setting unit 204, the partial Image extract- 
ing unit 206, the distinctive parts detecting unit 208, and 
the infomnation receiving unit 210 have the same func- 
tions as the fourth embodiment, the explanation is omit- 

20 ted. 

[0112] The processing condition setting unit 240 sets 
an image processing condition based on main subject 
infonnation. The main subject infonnation noted here 
means infomnation about the position of a main subject 

25 in image data, for example. Conditions described in the 
following can be thought of as image processing condi- 
tions. For example, the processing condition setting unit 
240 may decide conditions to undertake an image 
processing, which emphasizes the tint of the colors for 

30 the main subject and tones down the tint of the colors 
for the rest of the obj ects. Moreover, the processing 
condition setting unit 240 may also set conditions, which 
undertakes an image processing to tone the best gra- 
dation for both the main subject and the other subjects. 

35 [0113] An image processor 242 undertakes image 
processing on an image based on the Image processing 
conditions set by the processing condition setting unit 
240. The processed image will be stored on the hard 
disc 394 in Fig. 6 and be displayed on the monitor 342. 

40 As thefourth embodiment, the optional device controller 
374 and the communication l/F section 380 may output 
the processed image through a recording medium or a 
network. 

[01 1 4] According to another embodiment, this image 
45 processing apparatus may be a digital camera, which 
detects a main subject and undertakes the best image 
processing. In this case, the structure of the digital cam- 
era Is as shown in Fig. 1 . 

[0115] A desired image can therefore be obtained 
so since image processing is undertaken in conditions suit- 
ed to a main subject detected with high precision. For 
example, different image processing can easily be un- 
dertaken for a main subject and other subjects. 
[0116] The present invention according to the sixth 
55 embodiment will now be explained in the following. The 
Image processing unit according to the present embod- 
iment is a laboratory system, which detects a main sub- 
ject and outputs an image in the best condition. Since 
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the structure of this laboratory system is the same as 
the fourth embodiment in Fig. 6, the explanation is omit- 
ted. 

[01 1 7] Referring now to Fig. 9, the functional block di- 
agram of the laboratory system is shown according to 
the present embodiment Since the image data input 
unit200, the depth distribution information receiving unit 
202, the range setting unit 204, the partial Image extract- 
ing unit 206, the distinctive parts detecting unit 208, and 
the infomnation receiving unit 210 have the same func- 
tions as the fourth and the fifth embodiments, the expla- 
nation is omitted. 

[0118] The output condition setting unit 250 sets con- 
ditions for outputting images based on main subject in- 
formation. The main subject information noted here 
means information about position of the main subject in 
image data, for example. The following conditions can 
be thought of as conditions for outputting images. It 
takes a little bit of time to display an image that has a 
large amount of data when a monitor displays an image. 
Therefore, the output condition setting unit 250 may, for 
example, set conditions to display the part of a main sub- 
ject by priority and to display parts other than the main 
subject at a certain delayed time afterthe foregoing. Fur- 
ther, when a moving Image Is transferred through a net- 
work and is displayed on a monitor on the networks, the 
output condition setting unit 250 may decide conditions 
to set the rate of transfer for the part of the main subject 
high and for the parts other than the main subject low. 
Furthermore, the output condition setting unit 250 may 
decide conditions to undertake a process for modulating 
the frequencies for emphasizing the sharpness of the 
main subject and weakening the sharpness of the parts 
other than the main subject. 

[01 1 9] When the laboratory system 300 prints out an 
image as a photo, the output condition setting unit 250 
may, for example, decide conditions to print based on 
the part of the main subject. Further, the output condition 
setting unit 250 may decide conditions to convert gra- 
dations differently between the parts of the main subject 
and the parts of the other subj ects, according to the 
dynamic range of the image outputs of a printer or a 
monitor. 

[0120] The image data output unit 252 outputs Image 
data based on conditions for outputting images. The im- 
age data output unit 252 corresponds to the print unit 
330 or the display unit 340 shown in Fig. 6. 
[0121] According to another embodiment, this Image 
processing apparatus may be a digital camera, which 
detects a main subject and outputs images In a best con- 
dition. In this case, the structure of the digital camera Is 
as shown In Fig. 1 . 

[0122] Images of a main subject and other subjects 
can therefore easily be output in different conditions 
since a main subject can be detected with high precision 
based on depth distribution information. 
[01 23] The seventh embodiment of th e present inven- 
tion will now be described. According to the seventh em- 



bodiment, the Image processing apparatus and the Im- 
age processing method achieved in the embodiments 
from the first to the sixth are provided in a form of a re- 
cording medium recorded program software for reallz- 
5 Ing the functions on hardware. In this case, the main 
CPUs 62 and 362 activate based on a program stored 
in at least any one of the non-volatile memories 66 and 
366. the main memories 68 and 368, the hard disc 394, 
and the CD-ROM 398. Electric computers such as a per- 
sonal computer and a work station, for example, may 
be used as hardware, whtoh operate based on this pro- 
gram. 

[01 24] Fig. 1 0 is a block diagram showing the system 
construction of the program according to the seventh 
embodiment. This program software is, for example, 
stored In a recording medium such as the CD-ROM 398 
and is supplied to users. Generally, program software is 
installed in the hard disc 394 from a recording medium. 
The program software is then read to the main memory 
368 or is read directly to the main memory 68 from the 
recording medium. The read program software Is exe- 
cuted by the main CPUs 62 and 362. 
[01 25] As apparent from Rg. 1 0, the program accord- 
ing to the Invention has a structure constituted by a com- 
bination of a plurality of program modules that have var- 
ious functions . This program Is generally comprised of 
an Image data input module 260. a depth distribution 
information receiving module 262, a range setting mod- 
ule 264, a partial image extracting module 266, a dis- 
tinctive parts detecting module 268, and an information 
receiving module 270. The image data input module 260 
is a program for inputting image data of an object. The 
depth distribution information receiving module 262 is a 
program for receiving depth distribution Information 
based on parallactic images. The range setting module 
264 Is a program for setting a partial searching range In 
image data. The partial image extracting module 266 is 
a program for extracting from image data, a part from 
many objects included in a partial searching range as a 
partial image, based on depth distribution information. 
The distinctive parts detecting module 268 is a program 
for detecting a distinctive part from a partial image. The 
infonnatlon receiving module 270 is a program for re- 
ceiving main subject Infomnation based on the detected 
distinctive part. 

[0126] The CD-ROM 398 as an example of a record- 
ing medium can store a part of the operation or all of the 
functions of the main CPUs 62 and 362. The CD-ROM 
can also store a part of the operation or all of the func- 
tions of the other devices. 

[0127] An optical recording medium such as a DVD 
or a PD, a magnetic recording medium such as a floppy 
diskette era mini disc (MD), an optical magnetic record- 
ing medium such as anMO, taping recording medium, 
and a non -volatile semi-conductor memory card can be 
used as a recording medium other than the CD-ROM 
398. The recording medium, which stores the above 
program is only used for msmufacturing an image 
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processing apparatus. It is obvious that manufacturing 
and seiling of those recording mediums as a business 
constitute an Infringement of the patent right based on 
the present application. 

[0128] in the embodiments described above, the 
searching range Is defined, just for example, by the 
range of the depth length. However, the invention Is not 
limited thereto or thereby. For example, the distance to 
the object may also be calculated on the basis of a range 
of parallax amount, a range of intensity of light reflected 
by an object as described herelnbelow, or a range of 
size of an object. 

[0129] By the way, as discussed above, the searching 
range is defined, as an example, as the depth length in 
the foregoing embodiments. However, the Invention is 
not limited thereto or thereby. The other ranges, for ex- 
ample, a range of parallax, a range of intensity of light 
reflected from an object, a range of size of an object and 
the like may also be employed as a searching range for 
measuring the distance from the object. 
[0130] Further, in the embodiments described above, 
the depth distribution information receiving unit 202 is 
of a type called a passive type in which the depth distri- 
bution Infomiation Indicating the distances to the re- 
spective parts of the object on the basis of the parallactic 
Image input from image data input unit 200. However, 
the invention is not limited thereto or thereby. For exam- 
ple, the depth distribution information receiving unit 202 
may be of a type called active type in which a light source 
Irradiates light to an object and a distance between the 
light source and the object is measured on the basis of 
the intensity of the light reflected by the object. Some 
exsmriples employing the active type of the distance 
measuring system are described with reference to Fig- 
ures 11 through 14. 

[0131] Because the distance measuring apparatus 
and method of the active type is disclosed In great detail 
In Japanese Patent Applications Nos. 2000-37771 and 
2000-291 623, merely the outline of which is described 
in the following. 

[0132] Rg. 11 is a schematic view showing a distance 
measurement system of an active type according to a 
first example. 

[0133] The system includes a first light source 406 
and a second light source 408 performing as point 
sources which Irradiate light having the same wave- 
length. The first light source 406 and second light source 
408 are distanced from an object 402 by distances R1 
and R2, respectively. The first light source 408 is dis- 
tanced from the second light source 408 by a length L. 
[0134] Rrst. the light irradiating from the first light 
source 406 is reflected by a region 404 of the object 402, 
and the reflected light Is captured by a camera 405. After 
a predetermined time, the second light source 408 Irra- 
diates a light toward the object 402. and the reflected 
light is captured by the camera 405. The Intensity of both 
the lights irradiating from the first light source 406 and 
second light source 408, the intensity of both the lights 
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captured by the camera 405, a ratio of the intensity of 
the reflected lights and the distance L between the light 
sources are calculated with a following equation 1 . 

R1 = L/{ (W1/W2 . 11/12)^^ -1} (1) 

where R1 is a distance between the first ilght source 406 
and the region 404 of the object 402, L is a distance 
between the first Ilght source 406 and second light 
source 408, W1 and W2 are the intensity of lights re- 
flected by the region 404 of object 402, respectively, and 
II and 12 are the intensity of lights irradiating from the 
first light source 406 and second ilght source 408, re- 

^5 spectively. 

[01 35] As described above, the distance between the 
light source and the object can be measured by Irradi- 
ating two lights having the same wavelength at a differ- 
ent timing. On the other hand, however, the distance can 

^ also be measured by in-adlating two lights having differ- 
ent wavelength from each other at the same time and 
detecting the reflected lights from the object. 
[01 36] Fig. 1 2 Is a schematic top view showing a dis- 
tance measurement system of an active type according 
to a second example. 

[01 37] In this example, a first light source 41 OA and a 
second Ilght source 41 OA in^adlate lights at the same 
time. Those lights pass through a 41 2A and a 41 2B, re- 
spectively, to thereby generate a light having a wave- 
length AA and a light having wavelength X B and XC, 
respectively, and illuminate the region 404 of the object 
402. The reflected lights are focused by an optical lens 
420 and subsequently enter a beam splitter unit 430 
constituted by prisms in which the lights are split into 
three different lights having wavelength AA, XB and XC, 
respectively. Light receiving units 440A, 440 B and 440C 
receive the respective Ilght as shown in Fig. 12. The 
lights received by the light receiving units are processed 
by a processing unit 460 having a light intensity detect- 
ing section not shown in the figure drawings. The dis- 
tance from the light source and the region 404 of the 
object 402 is then measured by detecting the Intensity 
of the three different lights each having a different wave- 
length. The process of measuring the distance is sub- 
stantlally the same as that discussed above for the first 
example. 

[01 38] Fig. 1 3 Is a schematic top view showing a dis- 
tance measurement system of an active type according 
to an arrangement of the second example illustrated in 
Rg. 12. 

[0139] A first light source 41 OA and an optical filter 
41 2A produce a light having an Infrared wavelength XA 
while a second light source 41 OA and an optical filter 
41 2B do a light having two Infrared wavelength XB and 
AjC, and those lights are Irradiated toward the region 404 
of the object 402. On the other hand, the object 402 also 
receives visible, available ilght from the sun or a flash 
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light device. On the other hand, a beam split unit 430B 
constituted by prisms receives the reflected lights from 
the region 404 of the object 402 and then splits those 
lights Into three lights, XA, 7J3 and XC, and a visible light. 
The 440A, 440B and 440C receives the respective lights 5 
split by the beam split unit 430B. The distance from the 
light source and the region 404 of object 402 is then cal- 
culated by detecting the intensity of the three different 
lights each having a different wavelength. The process 
of measuring the distance is substantially the same as 
that discussed above for the first example. 
[0140] According to the first and second examples 
along with these arrangements shown in Figs. 11 
through 13, the distance measuring systems employ, by 
way of example, two different light sources. However, 
three or more light sources may be equipped in orderto 
enhance the accuracy or speed of the measurement 
[0141] Rg. 1 4 Is a chart for explaining the concept of 
the distance measuring system according to a third ex- 
ample of the present invention. 

[01 42] As shown in the drawing, an object 470 A is dis- 
tanced from a startpoint 432 for the distance measure- 
ment by a distance D(A) while an object 470B is dls- 
tancedfrom the the startpoint 432 forthe distance meas- 
urement by a distance D(B). Now a light beam emitting 
from the startpoint scans a plane perpendicular to a line 
connecting the startpoint 432 and the object 470A at the 
same scanning speed V(A) . During this scanning, a 
scanning speed V(B) of the light beam for scanning a 
plane perpendicular to a line connecting the startpoint 
432 to the object 470B is in proportion to a distance D 
(B) from the startpoint 432 to the object 470B. On the 
other hand, as the light beam scans at the save speed 
V(A), the time required for scanning a target point of the 
object 470B Is IninverseproportiontoadistanceD(B). 
Therefore, the greater the distance D (B) , the smaller 
the amount of integrated intensity S (B) of light beam 
irradiating the object 470B during the scanning period. 
On the other hand, a peal< intensity P(B) of light beam 
irradiating the object 470 B does not depend on the dis- 
tance D(B) provided that the light beam is a pure parallel 
beam. For the facts set forth above, accordingly, the dis- 
tance D(B) can be measured out by detecting the 
amount of Integrated intensity S (B) of light beam and 
the peak Intensity P(B) and also referring to a lookup 
table which stores previously the relations of the dis- 
tance D(B) and the ratio of the amount of integrated in- 
tensity S(B) of light beam and the peak intensity P(B), 
[01 43] Next, an eighth embodiment of the present in- 
vention will now be described. Fig. 15 is a functional 
block diagram of the digital camera according to the 
eighth embodiment of the invention. In Rg. 8, like parts 
and components are designated by the same reference 
signs as those shown In Fig. 2 which concems the first 
embodiment. 

[01 44] In the first embodiment, when the range setting 
unit 204 searches a main object, the partial searching 
range is automatically determined to search merely a 



depth length In which the main object would likely exists 
under a presupposition that the main object is usually a 
person or persons and that the main object usually 
stands at the frontmost position In the Image close to 
camera. The eighth embodiment, however, is provided 
with functions which is capable of being applicable to 
unusual pictures of which the main object is not a person 
or persons but a bird, flowers, or a landscape. 
[01 45] As shown In Fig. 1 5, first, an operator of a dig- 
ital camera inputs a desired operation mode through a 
mode selecting section 501 . The desired mode may be 
that, for example, the main object is a wild bird perching 
on a top of a tree far distanced from the camera or one 
uncommon flower among many flowers in a flower gar- 
den in the middle thereof. In this case, the operator man- 
ually sets a selection dial for main object and a distance 
dial to a desired mode at the mode selecting section 
501 . The operation mode which is selected through the 
mode selecting section 501 is input to a range setting 
unit 504. The range setting unit 504 determines a partial 
searching range In accordance with the given operation 
mode. 

[0146] Next, a partial image extracting unit 506 de- 
tects a target part to be included In the main object within 
the depth length range detemrilned by the range setting 
unit 504. As information with respect to the object In- 
tended by the operator is also transmitted to the partial 
image extracting unit 606, the partial image extracting 
unit 506 can periorm the searching operation accurately 
at a high speed. This Is because that, for example, if a 
flower Is the intended object, the target part may be a 
petal above leaves, whereas the target part may be an 
eye and a bill in front of the eye in a case where a bird 
Is an Intended object. The partial Image extracting unit 
506 thus determines the candidates of the target part of 
the object on the basis of Information included in the op- 
eration mode the operator selected. 
[0147] The candidate of the main object detected by 
the partial image extracting unit 506 is displayed on a 
display unit 502. if applicable, the candidate of the main 
object may be framed with a white line, or the white line 
may be blinked, so that the operator can readily recog- 
nize which object is now selected as a main object while 
viewing the screen of the display unit 502. If the candi- 
date of the main object Is not the one the operator In- 
tended, he or she operates a dial or button of a main 
object candidate changing section 503 to move to the 
next candidate. The operation repeats until the desired 
object is selected. 

[0148] According to the eighth embodiment, there is 
no possibility of misldentiftoation of the main object since 
the operator selects the main object white looking at the 
display unit. In particular, this system Is effective in a 
case where the main object Is not a person or persons. 
[0149] As described above, since objects having dif- 
ferent depth positions are excluded from a searching 
target, the possibility of recognizing the objects having 
different depth positions as one object in detecting a 
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main subject can be decreased. Further, since only the 
depth position has a high possibility of existence as a 
main subject as a target for searching, the time for de- 
tecting a main subject can be shortened. The image 
processing apparatus according to the present embod- 
iment can therefore detect a main subject with high pre- 
cision and effectively. 

[0150] Further, according to the first embodiment, an 
appropriate image can be obtained since the camera 
tal<es a photograph in conditions suited to a main sub- 
ject detected with high precision. 
[0151] Furthermore, according to the second embod- 
iment, the camera divides the depth range that includes 
all of the objects, and sets a plurality of partial searching 
ranges. Then a main subject is detected from ail of these 
partial searching ranges. Therefore, ail of the main sub- 
jects can be divided up and detected without misunder- 
standing when a plurality of main subjects are included 
in image data. 

[0152] Moreover, according to the third embodiment, 
a desired image can be obtained since the time when a 
main subject detected with high precision becomes a 
prescribed condition Is set as a photographic timing. A 
desired image can also be obtained without missing a 
photographic timing since a main subject can be detect- 
ed effectively and quickly. 

[0153] Further, according to the fourth embodiment, 
a desired image with high precision can be selected 
since a main subject detected with high precision is set 
as a target when deciding selection of an image. 
[0154] Furthermore, according to the fifth embodi- 
ment, a desired image can be obtained since an image 
processing is undertal^en in conditions suited to a main 
subject being detected with high precision. For example, 
different image processing can easily be given to both 
the main subject and the other subjects. 
[0155] Moreover, according to the sixth embodiment, 
images of the main subj ect and the other subj ects can 
easily be output in different conditions since a main sub- 
ject can be detected with high precision based on depth 
distribution information, 

[0156] Still further, according to the eighth embodi- 
ment, there is no possibility of misidentification of the 
main object because the operator of the digital camera 
selects the main object while viewing same on the 
screen of the display unit. 

[0157] As is obvious in the explanation described 
above, a main subject can be detected from an image 
according to the present invention. 
[0158] Although the present invention has been de- 
scribed by way of exemplary embodiments, It should be 
understood that those skilled in the art might make many 
changes and substitutions without departing from the 
spirit and the range of the present invention, which is 
defined only by the appended claims. 



Claims 

1 - An image processing apparatus, comprising an im- 
age data input unit for inputting Image data of an 
5 object, CHARACTERIZED IN THAT the apparatus 

further comprises: 

a range setting unit for restricting a searching 
range in the depth direction for searching a 
10 main object from the object using the input im- 

age data; and 

a partial image extracting unit for extracting 
from said image data, the part of an object in- 
cluded in said searching range as a partial Im- 
15 age. 

2. The image processing apparatus as claimed in 
claim 1, CHARACTERIZED IN THAT said Image 
data input unit comprises a means for inputting a 

20 parallactic image, which picks up an image of said 
object from a plurality of different perspectives, and 
the image processing apparatus further comprises 
a depth distribution information receiving unit for re- 
ceiving said depth distribution information based on 

25 said parallactic image. 

3. The Image processing apparatus as claimed in 
claim 1 , CHARACTERIZED IN THAT the apparatus 
further comprises a main subject detecting unit for 

30 detecting a main subject from said partial image and 
receiving main subject information. 

4. The image processing apparatus as claimed in 
claim 3, CHARACTERIZED IN THAT said main 

33 subject detecting unit comprises: 

a distinctive parts detecting unit for detecting 
from said partial image, a distinctive part, which 
should be included in said main subject; and 
40 an infomriation receiving unit for receiving said 

main sub] ect information based on the position 
of said distinctive part detected by said distinc- 
tive parts detecting unit. 

45 5. The image processing apparatus as claimed In 
claim 3, CHARACTERIZED IN THAT: 

said range setting unit comprises a means for 
setting a first searching range and a means for 

50 setting a second searching range, which differs 

from said first searching range; 
said partial image extracting unit comprises a 
means for extracting the part of said object in- 
cluded in said first searching range as a partial 

55 image and a means for extracting the part of 

said object included In said second searching 
range as a second partial image; and 
said main subject detecting unit comprises a 
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means for detecting said main subject from said 
first partial image and a means for detecting 
said nnain subject from said second partial im- 
age when said main subject Is not detected 
from said first partial image. 

6. The Image processing apparatus as claimed in 
claim 5, CHARACTERIZED IN THAT said range 
setting unit sets a predetemilned depth length as 
said first searching range and sets the depth length 
contiguous to said firstsearching range as said sec- 
ond searching range. 

7. The image processing apparatus as claimed In 
claim 5, CHARACTERIZED IN THAT said range 
setting unit sets the depth length, which includes an 
independent object existing at the nearest distance, 
as said first searching range based on said depth 
distribution infomnation and sets the depth length 
contiguous to said firstsearching range as sard sec- 
ond searching range. 

8. The Image processing apparatus as claimed in 
claim 3, CHARACTERIZED IN THAT: 

said range setting unit divides the depth range 
that includes said object in said image data 
based on said depth distribution Information 
and sets a plurality of said searching ranges ; 
said partial image extracting unit extracts from 
said Image data, the part of said object included 
In each of a plurality of said searching ranges 
as a plurality of said partial image; and 
said main subject detecting unit detects said 
main subject from each of a plurality of said par- 
tial Image and receh/es a plurality of said main 
subject information. 

9. The Image processing apparatus as claimed In 
claim 3, CHARACTERIZED IN THAT 

said range setting unit sets a different said 
searching range when said main subject is not 
detected In said searching range set by said 
range setting unit; 

said partial image extracting unit extracts again 
said partial image; and 

said main subject detecting unit detects again 
said main subject. 

10. The Image processing apparatus as claimed in 
claim 3, wherein said image data input unit is an 
Image capturing unit for picking up an image of said 
object, CHARACTERIZED IN THAT the apparatus 
further comprises: 

a photographic condition setting unit for setting 
a photographic condition based on said main 



subject Information; and 
an image capturing control unit for controlling 
imaging by said image capturing unit based on 
said photographic condition. 

5 

11. The image processing apparatus as claimed in 
claim 3, CHARACTERIZED IN THAT the apparatus 
further comprises: 

10 a processing condition setting unit for setting 

an Image processing condition based on said 
main subject information; and 
an Image processor for processing said image 
data based on said Image processing condl- 

IS tlon. 

12. The image processing apparatus as claimed in 
claims. CHARACTERIZED IN THAT the apparatus 
further comprises: 

20 

an output condition setting unit for setting an 
Image outputting condition based on said main 
subject Infomnation; and 
an image data output unltf or outputting said im- 
25 age data based on said image outputting con- 

dition. 

13. The image processing apparatus as claimed in 
claim 3, CHARACTERIZED IN THAT the apparatus 

30 further comprises: 

a selecting condition storing unit for storing a 
predetermined selecting condition of which 
said main subject should satisfies; and 
35 an image selecting unit for selecting from a plu- 

rality of said image data input by said image da- 
ta input unit, a desired image data, which In- 
cludes a desired main subject satisfying said 
selecting condition. 

40 

14. The Image processing apparatus as claimed in 
claim 3, wherein said image data input unit is an 
image capturing unit, which captures an image of 
said object based on a photographic timing signal, 

45 CHARACTERIZED IN THAT the apparatus further 
comprises: 

a timing condition storing unit for storing a pre- 
determined photographic timing condition relat- 
50 ing to said main subject; and 

a timing signal output unit for outputting said 
photographic timing signal to said image cap- 
turing unit when said main subject satisfies said 
photographic timing condition. 

55 

15. The Image processing apparatus as claimed in 
claim 1, CHARACTERIZED IN THAT the apparatus 
further comprises a distance measuring unit which 



15 



29 



EP 1 102 210 A2 



30 



compiises: 

a light Irradiating section for irradiating liglit to 
said object; and 

a calculating section for calculating a distance 
from said light irradiating section to said object 
on the basis of the intensity of light reflected 
from said object, and 

THAT said distance measuring unit outputs a 
distance Infomnatlon to said range setting unit. 

16. The image processing apparatus as claimed in 
claim 15. CHARACTERIZED IN THAT said light ir- 
radiating section comprises a first light source irra- 
diating light having a first wavelength and a second 
light source Irradiating lights having a second and 
a third wavelengths, and said first and second light 
sources are located at an optically different posi- 
tions. 

17. The image processing apparatus as claimed In 
claim 16, CHARACTERIZED IN THAT said first and 
second light sources irradiate lights at the same tim- 
ing. 

18. The image processing apparatus as claimed In 
claim 17, CHARACTERIZED IN THAT the appara- 
tus further comprises: 

an optically focusing section for focusing the 
lights irradiated by light in-adiating unit and re- 
flected from said object; 
a beam splitting section for optically splitting 
said reflected light Into a first light having said 
first wavelength, a second light having said sec- 
ond wavelength and a third light having said 
third wavelength; 

a light receiving section for receiving said re- 
flected lights spilt by said beam splitting sec- 
tion; and 

a light Intensity detecting section for detecting 
a light intensity of said first, second and third 
reflected lights received by said light receiving 
section, and 

THAT said calculating section calculates out a 
distance from said object on the basis of said 
light intensity detected by said light intensity de- 
tecting section. 

19. The Image processing apparatus as claimed in 
claim 16, wherein said light Inradlatlng section Irra- 
diates a light beam, CHARACTERIZED IN THAT 
the apparatus further comprises a light beam scan- 
ning section for scanning around a startpolnt said 
light beam toward a predetermined object within 
said object; and a reflected light beam detecting 
section for detecting light beams reflected by a tar- 
get of said object, wherein said calculating section 



comprises an Integrating section for calculating an 
amount of integrated intensity of said reflected light 
beams from said target of said object and a distance 
calculating section for calculating said distance on 
5 the basis of said integrated intensity and said peak 
intensity of said reflected light 

20. The Image processing apparatus as claimed in 
claim 1 , CHARACTERIZED IN THAT the apparatus 

10 further comprises a mode selecting unit for select- 
ing an operation mode con-esponding to an Intend- 
ed object, wherein said range setting unit restricts 
said searching range on the basis of said selected 
operation mode, and said partial image extracting 

IS unit extracts said partial image on the basis of said 
selected operation mode. 

21. The image processing apparatus as claimed in 
claim 1 or 20, CHARACTERIZED IN THAT the ap- 

20 paratus further comprises a display unit for display- 
ing said partial image extracted by said partial Im- 
age extracting unit and a changing section for 
changing a candidate of said objectselectedby said 
partial image extracting unit. 

25 

22. An Image processing method including a step of In- 
putting image data of an object, CHARACTERIZED 
IN THAT the method comprises steps of: 

30 setting a searching range, which defines a 

depth length for searching a main subject In 
said image data; 

extracting from said Image data, the part of an 
object Included In said searching range as a 
35 partial image, based on a depth distribution in- 

fomriatlon showing the distance between a 
camera and each part of said object included 
in said image data; 

detecting said main subject In said partial im- 
40 age; and 

receiving a predetermined main subject Infor- 
mation relating to the detected said main sub- 
ject. 

45 23. The Image processing method as claimed in claim 
22, CHARACTERIZED IN THAT: 

said setting is a first setting of a first searching 
range as said searching range; 

50 said extracting is a first extracting of the part of 

said object included in said first searching 
range, as a first partial image; and 
said detecting is a first detecting of said main 
subject In said first partial image, said detecting 

55 comprising: 

a second setting of a second searching 
range as said searching range when said 
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main subject Is not detected In said first 
partial Image; 

a second extracting of the part of said ob- 
ject Included In said second searcliing 
range as a second partial image; and 
a second detecting of said main subject in 
said second partial image. 

24. Hie image processing method as claimed In claim 
23, CHARACTERIZED IN THAT: 

said first setting is a setting of the depth length 
that includes an independent object existing at 
the nearest distance as said first searching 
range based on said depth distribution informa- 
tion; and 

said second setting is a setting of the depth 
length contiguous to said first searching range, 
as said second searching range. 

25. The image processing method as claimed In claim 
22, CHARACTERIZED IN THAT: 

said setting is a dividing the depth range that 
includes said object in said image data based 
on said depth distribution information and a set- 
ting of a plurality of said searching range; 
said extracting is a extracting from said image 
data, of the part of said object included in each 
of a plurality of said searching range as a plu- 
rality of said partial image; 
said detecting is a detecting of said main sub- 
ject from each of a plurality of said partial im- 
age; and 

said receiving is a receiving of said main sub- 
ject Infomiatlon relating to each of a plurality of 
the detected said searching range. 

26. The Image processing method as claimed In claim 
22, CHARACTERIZED IN THAT said Inputting is an 
Inputting at an image capturing unit, which captures 
an image of said object, comprising: 

deciding a photographic condition based on 
said main subject information; and 
controlling an image capturing at said Image 
capturing unit based on said photographic con- 
dition. 

27. The image processing method as claimed in claim 
22, CHARACTERIZED IN THAT the method further 
comprises steps of: 

setting an Image processing condition based 
on said main subject information; and 
processing said image data based on said im- 
age processing condition. 



28. The Image processing method as claimed in claim 
22. CHARACTERIZED IN THAT the method further 
comprises steps of: 

5 setting an image outputting condition based on 

said main subject information; 
outputting said image data based on said im- 
age outputting condition. 

10 29. The image processing method as claimed in claim 
22. CHARACTERIZED IN THAT the method further 
comprises steps of: 

storing a predeternilned selecting condition re- 
IS iating to said main subject; and 

selecting from a plurality of said image data in- 
put at said inputting, a desired image data that 
includes a desired main subject satisfying said 
selecting condition. 

20 

30. The Image processing method as claimed In claim 
22, wherein said inputting is a capturing of an image 
of said object based on a photographic timing sig- 
nal. CHARACTERIZED IN THAT said method fur- 

25 ther comprises steps of: 

storing a predetermined photographic timing 
condition relating to said main subject; and 
outputting said photographic timing signal to 
30 said image capturing unit when said main sub- 

ject satisfies said photographic timing condi- 
tion. 

31 . A recording medium, which records thereon a pro- 
35 gram for processing an Image and can be read by 

a computer comprising a module for inputting image 
data of a subject, CHARACTERIZED IN THAT the 
recording medium further comprises: 

a module for setting a searching range, which 
defines the depth length for searching a main 
subject in said image data; 
a module for extracting from said image data, 
the part of said object included in said search- 
ing range as a partial Image, based on depth 
distribution Infomnation showing the distance 
between a camera and said object Included in 
said Image data; 

a module for detecting said main subject from 
said partial Image; and 

a module for receiving predetermined main 
subject information relating to detected said 
main subject 
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